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MAINTAINABILITY  DATA  DECISION  METHODOLOGY  (MDDM) 


1.  INTRODUCTION 


Organizations  within  the  U.S.  Anny  [i.e.  Communications-Electronics  Command 
(CECOM)]  and  other  government  organizations  have  the  need  to  evaluate  Maintenance 
Manpower  requirements  for  systems  (i.e.  Power  Generators,  etc.)  where  fully  developed 
maintenance  data  is  NOT  available.  Maintenance  Manpower  requirements  are  computed  by 
multiplying  an  estimated  maintenance  ratio  (man-hours  per  usage)  by  an  annual  wartime  usage 
(AWU),  which  results  in  a  total  number  of  recommended  maintenance  man-hours  -  this  is  called 
direct  production  annual  maintenance  man-hours  (DPAMMH).  A  maintenance  ratio  (MR) 


A 

estimate  ( MR 


I 

7=1 


man  -  hours ; 


ophours  j 


)  is  based  on  a  random  sample  (without  replacement)  of  n  generators 


7=1 

from  a  finite  population,  where  the  pair  (man-hours  and  ophours)  are  associated  with  each 
generator.  Anny  organizations  need  to  know  how  much  maintenance  ratio  data  needs  to  be 
collected  until  the  sample  can  be  used  to  generate  maintenance  manpower  requirements.  The 
DPAMMH  will  reside  in  the  Army  Maintenance  Manpower  Data  Base  (AMMDB). 


Anny  Materiel  Systems  Analysis  Activity  (AMSAA)  developed  a  maintainability  data 
decisioning  methodology  (MDDM)  which  determines  if  enough  sample  maintenance  data  exists 
in  order  to  infer  the  true  fleet  MR.  This  will  allow  the  Army  to  make  manpower  requirement 
determinations.  The  true  fleet  MR  is  unknown  but  is  defined  to  be  the  sum  of  the  man-hours 
divided  by  the  sum  of  the  operating  hours  for  all  of  the  Army  generators  in  the  fleet  at  the  end  of 
useful  life.  Here  is  what  the  sample  of  maintenance  data  looks  like: 


Table  1.  Sample  Maintenance  Data. 


serial  # 

man-hours 

operating 

hours 

visits 

1 

mh(l) 

h(l) 

v(l) 

2 

• 

Mh(2) 

• 

h(2) 

• 

V(2) 

• 

• 

• 

n 

• 

• 

Mh(n) 

• 

• 

H(n) 

• 

• 

V(n) 

Along  with  the  man 
hours  at  each  visit. 
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It  is  impossible  to  collect  maintenance  data  on  all  generators  in  the  fleet  over  their  useful 
lives.  Even  the  Army  could  collect  on  all  of  the  generators;  a  manpower  decision  is  needed  way 
before  these  generators  reach  the  end  of  useful  life. 

The  Army’s  main  goal  is  to  make  sure  enough  sample  data  exists  to  infer  the  fleet  MR  - 
ultimately  they  want  to  be  sure  enough  manpower  can  support  the  mission,  without  wasting  too 
much  manpower.  To  assure  enough  manpower  exists,  MDDM  detennines,  through  a  coverage 
validation  model,  that  enough  data  exists  to  make  an  accurate  inference.  Additionally,  to  assure 
that  not  too  much  manpower  exists,  MDDM  determines  if  enough  data  exists  to  make  a  precise 
inference,  measured  by  the  size  of  the  inference. 

This  paper  discusses  and  develops  the  MDDM  methodology  which  aids  and  facilitates 
the  manpower  decisioning  process.  To  illustrate  the  process,  a  sample  of  notional  generators  is 
used. 


MDDM  uses  a  systems  aging  model,  generator-to-generator  variation  models,  two 
stochastic  inferencing  and  four  coverage  validation  models  using  nonparametric  &  parametric 
bootstrapping,  percentile  method  with  bias  correction  &  acceleration  using  jackknifing,  Monte 
Carlo  simulation,  and  other  stochastic  modeling  techniques  &  processes. 
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2. 


DEVELOPMENT  OF  METHODOLOGY  &  APPLICATION 


Sample  Data  and  Assumptions 

Let’s  take  a  random  sample  (without  replacement)  of  n  generators  (man-hours  [mh]  and 
operating  hours  [ophours]  for  each  vehicle)  from  a  fleet  of  generators.  Here  is  a  notional  sample 
of  14  generators: 

Table  2.  Generator  Sample. 


mh 

ophours 

visits 

18.1 

1114 

7 

23.7 

396 

5 

9 

268 

3 

10.3 

212 

3 

9.7 

789 

3 

6.1 

392 

3 

6.9 

112 

2 

4.8 

328 

2 

9.6 

560 

1 

3 

220 

1 

3.3 

220 

1 

3.1 

327 

1 

2.3 

357 

1 

8.8 

104 

1 

Total 

118.7 

5399 

34 

MR  = 

0.0220 

_ 

A  visit  is  defined  to  be  when  a  generator  was  taken  out  of  operation  to  be  repaired  for 
whatever  reason.  MDDM  assumes  ophours  between  visits  for  each  unit  are  assumed  to  be 
exponentially  distributed.  Hence  “no  aging”  for  each  generator.  To  test  this  assumption,  we 
analyze  each  generator  with  at  least  15  visits  by  using  a  systems  aging  model  [1].  This  aging 
model  is  based  on  the  power  law,  which  states  that  the  expected  number  of  visits  by  time  t,  i.e.  N 
(t),  is  governed  by  the  power  law  relationship: 

N(t)  =  ktPpL 

Where  X  ,  J3PL  ,  t>0;  X  is  the  scale  parameter,  and  J3PL  is  the  shape  parameter. 
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If  this  “no  aging  per  generator”  assumption  was  not  true  then  it  would  be  impossible  to 
plan  for  the  correct  manpower  because  the  manpower  requirement  would  either  be  constantly 
rising  or  falling. 

The  average  number  of  man-hours  per  visit  is  assumed  to  experience  no  generator-to- 
generator  variation,  and  is  modeled  with  the  best  fitting  distribution.  For  this  sample,  man-hours 
per  visit  are  best  fit  by  using  a  truncated  lognonnal  distribution,  based  on  a  comparison  of  Chi- 
Square  tests  results  (the  smaller  the  Chi-Square  statistic  the  better  the  fit). 

Generator  to  Generator  variation  Models 


The  variation  in  true  visit  rate  from  generator  to  generator  is  determined  based  on  the 
parent  population  of  the  true  visit  rates.  If  this  parent  is  known  to  be  a  Gamma  prior  distribution 
then  we  will  utilize  the  parametric  empirical  bayes  (PEB)  approach  [2]  to  model  the  true  visit 
rate  ( /f  )  variation  from  generator  to  generator. 

These  ’s  are  assumed  to  be  statistically  independent  realizations  of  a  Gamma 
distribution  with  parameters  a  and  (3  with  mean  a  /  P .  Two  methods  are  used  to  estimate  these 
parameters  using  the  PEB  approach:  Method  of  Moments  and  the  Marginal  Maximum 
Likelihood  (ML).  Refer  to  the  Appendix  of  [2]  for  more  detail  on  these  methods. 

The  weighted  moment  estimators  for  a  and  /?  become  [2]: 


(A„2)iy 

_ M _ 

N  _  _ 

2>,<m.=  -0-aa 


(i) 


and 


P  =  a/Aw. 

The  likelihood  equations  are  [2]: 

/?i>j /(',+«] 

a  = - - 

N-p±m]+p)\ 

7=1 


(2) 


(3) 


and 


4 


(4) 


Ann(/?)  +  X  £  — -fjln(tj+j3)  =  0 

7=1  1=0  &  + 1  7=1 

These  equations  (3)  and  (4)  must  be  iteratively  solved  to  obtain  the  ML  estimates  a  and  /? . 

The  estimates  for  a  and  /?  from  the  Weighted  Method  of  Moments  (1)  and  (2)  may  be  used  as 

starting  values  for  the  iteration  routine  to  solve  the  likelihood  equations. 

If  the  parent  distribution  for  the  true  visit  rates  in  unknown  then  we  will  utilize  the 
nonparametric  technique  called  empirical  bayes  (EB)  approach  [3]  to  model  the  true  visit  rate 
( Aj )  variation  from  generator  to  generator. 

The/f  ’s  are  generated  by  nonparametrically  bootstrapping  [4]  a  set  of  n  k’s,  where  the 

k’s  are  the  Lemon  Krutchkoff  (1969)  iterated  estimators  repeated  n  times  [3].  Nonparametric 
bootstrapping  is  defined  to  be  sampling  with  replacement  from  the  original  set  of  n  k’s.  For 
example,  if  { Al,A2,...,An  }  is  the  original  set  of  n  k’s,  then  the  bootstrap  set  of  n  k’s, 

{ A2,A2,...,An  }  is  one  possible  bootstrap  sample.  There  are  a  total  of  n"  total  possible  bootstrap 
samples. 

Based  on  our  notional  example  of  14  generators,  we  see  that  the  true  visit  rates  are  best 
modeled  with  the  EB  approach  since  the  Poisson  data  does  not  fit  the  PEB  assumptions  of  a 
Gamma  prior  distribution  on  the  true  visit  rates. 


Inferencing  Models  and  Precision 

Now,  we  want  to  use  the  sample  of  14  generators  and  all  of  the  information  that  was 
gathered  (i.e.  mh  distribution  and  generator  to  generator  variation)  to  infer  the  fleet  MR.  We 
want  to  be  90%  (i.e.  the  decision  maker  assesses  the  confidence  level)  certain  that  we  have  an 
adequate  MR  estimate.  Adequate  implies  estimate  is  at  or  above  the  fleet  MR.  To  achieve  this 
goal  we  need  to  construct  a  1 -sided  90%  upper  confidence  bound  (UB)  for  the  fleet  MR.  This 
will  assure  us  that  we  have  enough  manpower. 

In  order  to  create  this  1  -sided  UB  stochastic  model,  we  build  a  sampling  distribution  of 

MR  ’s  using  the  original  sample  information  and  Monte  Carlo  simulation  [5].  The  first  step  is  to 
obtain  14  k’s  from  the  Gamma  distribution,  if  the  Poisson  data  fit  the  PEB.  Note,  if  the  data  did 
not  fit  the  PEB,  then  we  use  the  EB  and  we  would  bootstrap  the  Lemon  Krutchkoff  k’s  to  obtain 
a  bootstrap  sample  of  14.  Next,  we  perform  a  Poisson  process  on  each  generator  using  /L;  and 

ophourj  to  obtain  the  #  of  visits,  v j  -  visits  are  a  realization  of  a  Poisson  random  variable  with 
mean  =  x;  *  ophourj .  Finally  we  draw  v .  man-hours  for  each  generator  from  the  fitted 
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truncated  lognonnal.  Now  we  obtain  MR  1 .  Repeat  this  entire  step  B  (say  500+)  times  to  obtain 
B  estimates  (bootstrap  replications)  of  MR. 

Now  we  apply  the  Bias  Corrected  and  Accelerated  (BCa)  method  [4]  to  this  distribution 
of  500  estimates  of  MR..  The  BCa  method  is  basically  an  adjustment  (for  non-nonnal  data)  to 
the  percentile  points  of  the  Percentile  Method.  The  “BC”  adjusts  when  the  mean  and  median  are 
not  equal  and  the  “accelerated  (a)”  variance  stabilizes  the  distribution  -  hence  both  together  try 
to  normalize  the  distribution. 

A  (<*,) 

Let  MR  indicates  the  100*  as  th  percentile  from  the  500  estimates  of  MR.  This 
represents  the  percentile  method.  The  upper  bound  using  the  BCa  method  is  given  by: 

A  (<*2)  2  I  _(!-<*! ) 

MR  ,  where  a,  =  ®(z0  +  ttt~  U-gK)  (5) 

1  -a(z0  +  r  "0 

Here  ®(*)  is  the  standard  nonnal  cumulative  distribution  function  and  z  '  a'  ]  is  the 
100*  (1  -  as)  th  percentile  point  of  a  standard  nonnal  distribution.  For  example  z* 95)  =  1 .645  and 
0(1.645)  =.95. 

The  value  of  BC  is  derived  by  the  proportion  of  bootstrap  replications  that  is  less  than  the 
original  estimate  MR  .  Here  is  that  value: 


„  t  #{MRb  <MR} 
z0= O  (— - - - ~) 


(6) 


The  acceleration  constant  is  computed  using  jackknifing.  Here  it  is: 

n  3 

YjMRn  —  MR(,)) 


6{J(MR(*)-MR,0)  }3/2 

i= 1 

Where  MRu)  is  computed  using  the  original  sample  with  the  ith  generator  deleted  -  this 
is  jackknifing.  MR{ *>  is  the  weighted  average  of  {MRa^MRa), . ,MR0,)} 
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Below  is  a  flowchart  of  the  entire  process  for  the  stochastic  1  -sided  UB  inferenceng 

model. 

Table  3.  Inferencing  Model  Flowchart. 


I.Sampeofmantenancedatafor  , 
n  gene'ators  (mb,  optiours,  visits) 


II.  fit  mb  data  (e.g.runcated  ogno-ma  | 
Y 


III.  Does  Posson  data 
fit  PEB? 


"Ni 


Ert  mate  pa'amete'Sfor  Gamma 
dstn  but  on  of  true  vs  t -ates. 


IV.  ^ 

true  visit 

•ate  opboui 


Est  matenfue  vist  'Xes  using  EB. 


A 

X, 


vs  ts 


mh 


Poisson  random 
devotee 
wltn  mean 

U  =  V, 


mh. 

t- 


^"/^opkMtrs-  •WSl 


Draw  n  true  visit  rates  using  either  parametric  or 
nonparametric  bootstrapping. 


Draw  random  deviates  from  mh  distribution! 


V.  Reseat  step  lV500ftmes  and  obtain 

vl  usingthe  Percent  e  Method 
with  B  as  Correcton  &  Acre  eration  |BCa  Method] 
we  obtan  the  1-sided  9<3t  UBfor  Mt. 

Fo- details  of  this  BCa  Method ,  'eferto  : 

Eton.  aandTibshirari  RJ  .  "An  ntroductontotne  Bootstap\  London UJC Chapman  &  HaiVCKX  1555 


10B4  +f-dependingon  BCa  Mrthod 
(i  .e.  co  rrected  fo  r  non  -normal 
sam  p  I  i  n  g  d  istr  ibu  tion  s .) 


So  using  our  sample  of  14  generators  and  this  inferenceing  model,  here  are  the 
stochastically  created  1 -sided  90%  UB’s  for  MR  and  the  two  metrics  that  are  used  to  derive  MR  : 

Table  4.  Infercing  Model  Results. 


Metric 

Mean 

UB 

Precision 

MR 

0.0246 

0.0319 

0.30 

Visit  Rate 

0.0071 

0.0086 

0.21 

Man-hours 

3.48 

4.17 

0.20 

Our  2nd  goal  is  to  make  sure  the  UB  isn’t  too  far  to  the  right  of  the  mean.  Let’s  define 
precision  to  be  the  percent  difference  from  the  mean  estimate  to  the  UB  (some  applications 
define  precision  to  be  the  absolute  difference).  Since  it  is  difficult  to  access  the  precision  for  a 
multi-variable  metric  like  MR,  we  examine  the  precision  for  the  two  single  metrics  that  derive 
MR,  since  we  have  experience  assessing  precision  for  these.  The  decision  maker  determines 
what  amount  of  precision  they  can  tolerate.  Based  on  past  experience,  20%  precision  was 
recommended  for  the  single  metric  applications. 
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As  you  can  see  from  the  inferencing  results,  precision  for  the  visit  rate  is  21%,  which  is 
above  20%.  Therefore,  we  do  not  have  enough  data  to  make  a  precise  inference  using  a  20% 
precision  decision  requirement. 

For  this  example  we  keep  collecting  data.  However,  if  enough  data  was  present  to 
conclude  an  acceptable  precision  then  we  would  need  to  make  sure  the  90%  1 -sided  Cl  is 
accurate.  To  validate  this  we  proceed  to  the  next  section. 

Validation  Coverage  Models  and  Accuracy 

In  order  to  accurately  build  this  1  -sided  UB  stochastic  model,  we  need  to  assure  that  the 
sample  has  “enough  (#  of  generators  and  ophours)”  data  to  achieve  the  90%  level.  To  validate 
this  accuracy  we  use  coverage  models. 

First  let’s  define  what  we  mean  by  coverage  and  accuracy.  Coverage  is  defined  to  be  the 
percentage  of  UB’s  that  are  greater  than  the  true  population  MR,  where  each  UB  is  constructed 
with  some  method  at  the  90%  confidence  level  for  a  given  random  sample  of  n  generators  from 
the  population.  In  other  words,  we  need  to  run  the  inference  method  discussed  in  the  prior 
section  500+  times  (500+  samples  drawn  from  a  parametric  or  nonparametric  population)  to 
obtain  500+  inferences  (i.e.  500+  UB’s).  These  500+  samples  are  not  to  be  confused  with  the  B 
(500+)  iterations  from  the  method.  Note,  the  500+  simulated  populations  are  built  based  on  the 
sample  information.  Then  we  determine  how  many  UB’s  are  greater  than  the  true  MR. 

Accuracy  is  just  a  convergence  rule  for  explaining  the  relative  error  of  a  1 -sided 
coverage.  The  rule  focuses  on  the  speed  at  which  the  relative  error  approaches  0.  Second  order 
accuracy  is  defined  as  the  actual  non-coverage  probability  intended  to  be  as  %  for  a  1  -sided 
(1  -  as )  %  Cl,  approaches  the  ideal  of  a,  %  with  error  proportional  to  1/n  [6].  First  order 

accuracy  would  approach  the  ideal  of  a  %  with  error  proportional  to  L  .  This  means  that  the 

A Jn 

relative  error  of  the  1 -sided  coverage  is  of  the  order  0(l/n)  for  second  order  accuracy  and  0(  -)=  ) 

•Jn 

for  first  order  accuracy.  BCa  is  2nd  order  accurate  since  it  adjusts  the  percentile  points  based  on 
the  non-nonnal  data.  The  percentile  method  is  only  1st  order  accurate  since  it  does  not  make  any 
adjustments  to  the  percentile  points. 

Using  our  sample  of  14  generators  and  a  validation  coverage  model,  we  obtain  an 
acceptable  coverage  of  87%,  when  we  ask  for  90%.  The  absolute  relative  error  was  within  our 
5%  requirement.  Therefore,  for  this  sample  it  is  appropriate  to  use  the  inferencing  model.  Note, 
that  we  run  this  validation  coverage  model  on  a  sample  by  sample  basis.  This  makes  sense  since 
we  want  to  make  sure  our  particular  sample  (#  of  generators  and  ophours)  will  produce  accurate 
results. 
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3.  CONCLUSIONS 


AMSAA  has  successfully  applied  and  illustrated  the  MDDM  process  to  a  notional  set  of 
generators.  It  can  be  shown  that  increased  generator  to  generator  variation  implies  more  sample 
data  is  needed  to  assure  accuracy  and  precision.  It  can  also  be  shown  that  decreased  number  of 
visits  implies  more  sample  data  is  needed  to  assure  accuracy  and  precision. 

MDDM  is  a  decisioning  process  that  detennines  if  enough  sample  data  is  present  to  make 
an  accurate  and  precise  inference  on  the  MR.  MDDM  is  NOT  a  sampling  problem  or  sequential 
problem  with  a  target  in  mind. 

Finally,  if  accuracy  and  precision  are  satisfied,  then  the  aging  risk  needs  to  be  considered 
before  any  the  final  decision  is  made.  MDDM  can  be  applied  to  other  Army  weapon  systems 
where  manpower  decisions  are  required. 
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